[Python] Detect destination of shortened, or "tiny" url

Posted by conradlee on Stack Overflow See other posts from Stack Overflow or by conradlee
Published on 2010-03-16T12:11:10Z Indexed on 2010/03/16 12:16 UTC
Read the original article Hit count: 401

Filed under:

python

|

url

|

redirection

|

tinyurl

I have just scraped a bunch of Google Buzz data, and I want to know which Buzz posts reference the same news articles. The problem is that many of the links in these posts have been modified by URL shorteners, so it could be the case that many distinct shortened URLs actually all point to the same news article.

Given that I have millions of posts, what is the most efficient way (preferably in python) for me to

detect whether a url is a shortened URL (from any of the many URL shortening services, or at least the largest)
Find the "destination" of the shortened url, i.e., the long, original version of the shortened URL.

Does anyone know if the URL shorteners impose strict request rate limits? If I keep this down to 100/second (all coming form the same IP address), do you think I'll run into trouble?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about url

mod_rewrite for clean URL doesn't convert the URL to clean URL (but it's accessible) [on hold]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Basically what I want to do is to convert this: http://localhost/jariungu/user_caleg.php?idCaleg2014=3 into this: http://localhost/jariungu/caleg/3 I have managed to make /jariungu/caleg/3 to direct to the original URL (as in, if I open that URL, it directs me to the appropriate page). The problem… >>> More
Tip/Trick: Fix Common SEO Problems Using the URL Rewrite Extension

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Search engine optimization (SEO) is important for any publically facing web-site. A large % of traffic to sites now comes directly from search engines, and improving your site’s search relevancy will lead to more users visiting your site from search engine queries. This can directly or… >>> More
mod_rewrite one url to another url without changing source url

as seen on Stack Overflow - Search for 'Stack Overflow'
Is it possible to do a mod_rewrite from one url to another without changing what appears in the address bar? Example: Source URL is http://domain1.com/news Target URL is http://domain2.com/news I want to render pages from http://domain2.com/news/ but have http://domain1.com/news appear in the address… >>> More
ASP.NET MVC without Url Rewriting/Pretty Url

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have hosting with IXWebHosting, but ISAPI_Rewrite is not enabled on their Windows Server. So I want to deploy my site without Url Rewriting/Pretty Url. Is it possible? Thanks in advance, Munim >>> More
Ant get task throws "get doesn't support nested resources element" error

as seen on Stack Overflow - Search for 'Stack Overflow'
The following ant xml should work according to documentation, but does not. Can anyone tell me if I'm doing something wrong. The get task should support the nested "resources" element in Ant 1.7.1 which is the version I'm using: … >>> More